Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as “automated assistants” (also referred to as “chatbots,” “interactive personal assistants,” “intelligent personal assistants,” “personal voice assistants,” “conversational agents,” etc.). For example, humans (which when they interact with automated assistants may be referred to as “users”) may provide commands, queries, and/or requests using spoken natural language input (i.e. utterances) which may in some cases be converted into text and then processed, and/or by providing textual (e.g., typed) natural language input. Some automated assistants may be able to “remember” simple facts when requested to do so by a user. For example, a user may request that an automated assistant remember the user's name, the user's spouse's name, the user's anniversary, etc. The user may later request the automated assistant to recall these facts. However, these simple facts tend to be stored in isolation, without much in the way of contextual information.